[SPARK-57681][SQL] Support dynamic table options for UPDATE by anuragmantri · Pull Request #56792 · apache/spark

anuragmantri · 2026-06-25T23:07:22Z

What changes were proposed in this pull request?

This PR add the dynamic table options syntax to UPDATE statements.

Why are the changes needed?

This is a continuation to adding options syntax in DMLs SPARK-49098

Does this PR introduce any user-facing change?

Yes, it adds new SQL syntax. For example:

 UPDATE catalog.db.table WITH (`k` = 'v' )
 SET status = 'archived' WHERE created_at < '2024-01-01'.

How was this patch tested?

Added tests to the following suites.

Parser (DDLParserSuite)
End-to-end (UpdateTableSuiteBase)
LogicalWriteInfo (InMemoryRowLevelOperationTable)

Was this patch authored or co-authored using generative AI tooling?

I used Claude Code (Claude Opus 4.8) to generate the code and tests and verified manually.

anuragmantri · 2026-06-25T23:09:37Z

@szehon-ho, tagging you since you made the INSERT counterpart :)

uros-b · 2026-06-26T11:33:30Z

          // don't rewrite as the table supports truncation
          d

        case r @ ExtractV2Table(t: SupportsRowLevelOperations) =>


Please consider this concern: silent option-drop on non-rewrite delete paths: for a table implementing both SupportsRowLevelOperations and SupportsDeleteV2 (e.g. Iceberg), a DELETE … WITH(...) WHERE can take the metadata-only/deleteWhere path (OptimizeMetadataOnlyDeleteFromTable / DataSourceV2Strategy), which has no options parameter; so the user's options are silently ignored. Same for the TruncatableTable truncate path. This would be a new user-visible "silently ignored" surprise, and no test currently covers this path.

Great catch. I looked at those paths. The options are indeed not being passed all the way. Fixing this requires DSv2 API changes something like this with default methods.

// SupportsDeleteV2 default void deleteWhere(Predicate[] predicates, CaseInsensitiveStringMap options) { deleteWhere(predicates); // default ignores options → fully back-compatible } // TruncatableTable default boolean truncateTable(CaseInsensitiveStringMap options) { return truncateTable(); }

I can make this change in this same PR or can do a follow up if DSv2 changes need separate PRs. Let me know what you think is the best way forward.

uros-b · 2026-06-26T11:34:27Z

-    | UPDATE identifierReference tableAlias setClause whereClause?                 #updateTable
+    | DELETE FROM identifierReference optionsClause? tableAlias whereClause?       #deleteFromTable
+    | UPDATE identifierReference optionsClause? tableAlias setClause whereClause?  #updateTable
    | MERGE (WITH SCHEMA EVOLUTION)? INTO target=identifierReference targetAlias=tableAlias


So this PR makes INSERT/DELETE/UPDATE accept WITH(...), but how about MERGE?

I intentionally did not add MERGE as the semantics are a bit different since MERGE statements have source and target relations, I'm still thinking about how options would look like there. I can file a separate JIRA and PR for MERGE later.

uros-b · 2026-06-26T11:35:36Z

        stop = 56))
  }

+  test("delete from table: with options") {


Please make sure to either update the SQL reference docs, or file a followup Jira ticket to do so later.

For sure. I took a look and looks like INSERT docs are also not updated. I will create a follow up JIRA to update all options related DMLS at once.

anuragmantri · 2026-06-30T21:44:16Z

@dongjoon-hyun - Could you please review this PR since you had reviewed its precedent (SPARK-49098) as well? Thank you!

dongjoon-hyun

Thank you for pinging me, @anuragmantri .

In general, given the code contribution, the claim of this PR title is too broad. When you make a PR, please revise the PR title by narrowing down as much as possible to the exact contribution scope.

We don't need to add MERGE here. Instead, I'd like to recommend to proceed DELETE FROM syntax in another JIRA issue after removing this PR because it could be misleading if we ignore options. For UPDATE syntax, IIUC, there is no silent ignoring code path.

Lastly, follow-up have a special meaning in the community. So, please don't say follow-up of something especially when the Fix Version is different.

If you narrow down the scope according to the above, I believe we can get a higher chance of merge to Apache Spark 4.3.0, @anuragmantri .

anuragmantri · 2026-07-01T20:33:30Z

Thanks for the valuable feedback @dongjoon-hyun. As you suggested, I have narrowed the scope of this JIRA and the PR to only UPDATE as it is the most unambiguous. I will work on the other DMLs and the docs on separate JIRAs.

This is ready for another round. Thanks!

dongjoon-hyun

What happens for the following queries?

UPDATE t WITH (...) ...
WHERE id IN (SELECT ... FROM t)

We don't consider these new options for analyzer relation caches, do we?

spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/RelationResolution.scala

Lines 238 to 242 in 972f696

    
           case CatalogAndIdentifier(catalog, ident) => 
        
             val key = toCacheKey(catalog, ident, finalTimeTravelSpec) 
        
             val planId = u.getTagValue(LogicalPlan.PLAN_ID_TAG) 
        
             relationCache 
        
               .get(key)

Given that, it seems that there are more corner cases with analyzer relation cache. Could you add more test coverage and verity them? Typically, Subquery and CTE are the corner cases we need to consider carefully, @anuragmantri .

dongjoon-hyun · 2026-07-01T22:03:37Z

    | fromClause multiInsertQueryBody+                                             #multiInsertQuery
    | DELETE FROM identifierReference tableAlias whereClause?                      #deleteFromTable
-    | UPDATE identifierReference tableAlias setClause whereClause?                 #updateTable
+    | UPDATE identifierReference optionsClause? tableAlias setClause whereClause?  #updateTable


Although this seems to follow read path, this is inconsistent with our existing INSERT syntax.

spark/sql/api/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseParser.g4

Lines 592 to 595 in 972f696

| INSERT (WITH SCHEMA EVOLUTION)? INTO TABLE? identifierReference tableAlias optionsClause? (BY NAME)?

REPLACE (WHERE | ON) replaceCondition=booleanExpression #insertIntoReplaceBooleanCond

| INSERT (WITH SCHEMA EVOLUTION)? INTO TABLE? identifierReference tableAlias optionsClause? (BY NAME)?

REPLACE USING identifierList #insertIntoReplaceUsing

UPDATE should follow INSERT style.

dongjoon-hyun · 2026-07-01T22:05:51Z

        stop = 70))
  }

+  test("update table: with options") {


Please use test prefix when it's possible.

- test("update table: with options") { + test("SPARK-57681: update table: with options") {

dongjoon-hyun · 2026-07-01T22:13:12Z

I finished the second-round review, @anuragmantri .

Please follow INSERT's syntax instead of SELECT. tableAlias should come first.
Please double-check RelationResolution class. Especially, subqueries and CTE cases. At least, this PR should add a test coverage to clarify what is supposed by this PR.
And, a minor comment about test prefixes.

uros-b reviewed Jun 26, 2026

View reviewed changes

dongjoon-hyun requested changes Jun 30, 2026

View reviewed changes

[SPARK-57681][SQL] Support dynamic table options for UPDATE

50072c3

anuragmantri force-pushed the delete-update-sql-options branch from 398cadc to 50072c3 Compare July 1, 2026 20:30

anuragmantri changed the title ~~[SPARK-57681][SQL] Support dynamic table options for DELETE and UPDATE~~ [SPARK-57681][SQL] Support dynamic table options UPDATE Jul 1, 2026

anuragmantri changed the title ~~[SPARK-57681][SQL] Support dynamic table options UPDATE~~ [SPARK-57681][SQL] Support dynamic table options UPDATE Jul 1, 2026

anuragmantri changed the title ~~[SPARK-57681][SQL] Support dynamic table options UPDATE~~ [SPARK-57681][SQL] Support dynamic table options for UPDATE Jul 1, 2026

dongjoon-hyun reviewed Jul 1, 2026

View reviewed changes

	case CatalogAndIdentifier(catalog, ident) =>
	val key = toCacheKey(catalog, ident, finalTimeTravelSpec)
	val planId = u.getTagValue(LogicalPlan.PLAN_ID_TAG)
	relationCache
	.get(key)

	\| INSERT (WITH SCHEMA EVOLUTION)? INTO TABLE? identifierReference tableAlias optionsClause? (BY NAME)?
	REPLACE (WHERE \| ON) replaceCondition=booleanExpression #insertIntoReplaceBooleanCond
	\| INSERT (WITH SCHEMA EVOLUTION)? INTO TABLE? identifierReference tableAlias optionsClause? (BY NAME)?
	REPLACE USING identifierList #insertIntoReplaceUsing

Uh oh!

Conversation

anuragmantri commented Jun 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

anuragmantri commented Jun 25, 2026

Uh oh!

uros-b Jun 26, 2026

Choose a reason for hiding this comment

Uh oh!

anuragmantri Jun 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

uros-b Jun 26, 2026

Choose a reason for hiding this comment

Uh oh!

anuragmantri Jun 27, 2026

Choose a reason for hiding this comment

Uh oh!

uros-b Jun 26, 2026

Choose a reason for hiding this comment

Uh oh!

anuragmantri Jun 27, 2026

Choose a reason for hiding this comment

Uh oh!

anuragmantri commented Jun 30, 2026

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

anuragmantri commented Jul 1, 2026

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

anuragmantri commented Jun 25, 2026 •

edited

Loading

anuragmantri Jun 27, 2026 •

edited

Loading

dongjoon-hyun commented Jul 1, 2026 •

edited

Loading